WIP: k8s lineages - #230

Phase 1 of the k8s training lineage integration: a `roar.backends.k8s`
backend that instruments plain batch/v1 Jobs submitted through
`roar run kubectl apply|create -f <manifest>`, gated by k8s.enabled.

- submit planning: pre-registers a GLaaS fragment session, rewrites the
  manifest (0600 prepared file + context sidecar), returns session_id so
  the shared submit finalizer reconstitutes after completion; degrades
  to an uninstrumented submit with a warning when GLaaS is unavailable
- manifest rewriter: wraps explicit container commands via a /bin/sh
  bootstrap that pip-installs the roar runtime and execs pod_entry,
  falling back to the original command on any bootstrap failure;
  injects the env/identity contract (Secret-backed session creds,
  downward-API pod identity, cluster-visible GLAAS_URL); appends the
  Secret document; fails actionably on missing explicit commands,
  generateName, or multiple Jobs
- pod_entry: roar init + roar run (preload) around the original
  command, then exports the recorded job as an ExecutionFragment with
  task identity pod_uid:container:completion_index:restart_attempt and
  streams it via the shared fragment transport (best-effort; training
  exit code always wins)
- host execution: kubectl submit, prepared-manifest cleanup, Job
  terminal-condition wait, submit job recorded with the ORIGINAL
  command and plan-generated parent uid so reproduce re-enters the
  backend and pod fragments link to the submit node
- reconstituter + K8S_FRAGMENT_LINEAGE_BACKEND merge k8s_task child
  jobs through the shared fragment lineage engine
- `roar k8s prepare` CLI for inspectable rewrites (no Secret embedded)
- shared roar.execution.fragments.export extracted from the OSMO bundle
  exporter (OSMO delegates, behavior unchanged)
- registry: k8s builtin (priority 95), config section, job-env marker

Verified: product-path e2e green on the KIND harness (4 tests: rewrite,
streaming, reconstitution into .roar/roar.db, secret hygiene) plus the
Phase-0 runtime smoke (7 total); 23 new unit tests; default gate 1993
passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Phase 2 of the k8s training lineage integration on top of the Job
backend: operator kind adapters, multi-pod distributed capture, and an
attach flow for fire-and-forget submits.

- manifest rewriter generalized to a WORKLOAD_KINDS adapter registry:
  batch/v1 Job, jobset.x-k8s.io JobSet (nested replicatedJobs pod
  templates), kubeflow.org/v1 PyTorchJob (per-role replica specs), and
  trainer.kubeflow.org TrainJob (wraps the spec.trainer command/env
  override; requires an explicit trainer command). Task names carry the
  replica role; terminal wait conditions unioned across kinds in the
  new workload_wait module
- pod identity node-index chain extended: JOB_COMPLETION_INDEX ->
  PET_NODE_RANK -> pod-level RANK (PyTorchJob v1)
- roar k8s attach WORKLOAD: recovers the parent job uid and Secret name
  from the cluster object's own env contract, resolves credentials
  (local key -> cluster Secret -> --session-file), optionally waits,
  records a local attach job under the recovered parent uid, and
  reconstitutes streamed fragments — the CI flow
- harness: bootstrap --with-jobset installs the JobSet controller
  (v0.12.0); new distributed e2e covers a 2-pod Indexed Job (shared
  fragment session, distinct completion-index identities, child-process
  capture, cross-pod artifact edge with topological step ordering),
  attach from a fresh project via the cluster Secret, and the same
  worker as a JobSet through the real controller

Verified live on KIND: 12/12 e2e (5 new distributed + 4 product path +
3 smoke); 21 new unit tests (65 total k8s-related); default gate 2003
passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Completes the remaining Phase-2 transport/capture slices.

Bundle fallback (GLaaS-less pods):
- k8s.bundle_dir config names a mounted shared volume, injected as
  ROAR_K8S_BUNDLE_DIR; pod_entry probes GLaaS reachability upfront and
  writes roar-fragments-<pod>.json bundles there when streaming is
  unavailable (the streamer swallows per-batch POST failures and
  reports "streamed" regardless — hence the probe; surfacing streamer
  failure counts is a noted follow-up). Training never fails on
  lineage-transport outages
- roar k8s ingest-bundles DIR merges a host-visible copy of the bundle
  volume through the shared fragment lineage engine (bundles.py)

Object-store I/O hooks (direct S3 is HTTP, invisible to the tracer):
- the k8s backend now ships a RuntimeImportAdapter; the existing
  roar_inject.pth/sitecustomize dispatch (activated by ROAR_WRAP=1,
  which pod_entry sets for the traced tree) patches
  botocore.client.BaseClient._make_api_call plus aiobotocore's async
  variant (covers s3fs/fsspec) to append S3 data-op events to
  ROAR_K8S_OBJECT_IO_FILE (object_io.py); hooks only record after the
  real call succeeds, never raise into user code, and no-op when the
  env is unset
- pod_entry folds the events into the exported fragment as s3:// refs
  with etag hashes, deduped last-wins per (mode, path)

Verified live on KIND: 14/14 e2e — new bundle-fallback test (black-hole
cluster GLaaS URL, bundle pulled off the node with docker cp, ingested;
note /var/tmp hostPath because KIND nodes mount /tmp as tmpfs which
docker cp cannot read) and MinIO S3 test (host-staged dataset, pod
get->put captured as job inputs/outputs with etag artifact hashes).
10 new unit tests; default gate 2013 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Closes the ranged-read fidelity gap that was previously the roar-proxy's
main advantage over the in-process hooks (the proxy itself is now
explicitly a Phase-3 opt-in webhook-injected sidecar for non-Python S3
clients — see the design doc's resolved open question 5).

- object_io: GetObject Range headers are parsed (fully-specified
  bytes=start-end pairs only) and recorded on events; the loader
  accumulates deduped ranges per (mode, path) across events while
  etag/size stay last-wins
- shared fragment path (additive): ArtifactRef gains an optional
  byte_ranges field with validating hydration, and the fragment merge
  engine now carries it into job_inputs/job_outputs.byte_ranges —
  previously fragments dropped ranges entirely (Ray/OSMO fragments
  gain the capability for free)

Verified live on KIND: MinIO e2e extended with a ranged GetObject,
asserting byte_ranges [[0,9]] on the s3:// input row; 14/14 e2e,
3 new unit tests, default gate 2015 passed, ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Completes Phase 2 of the k8s training lineage integration.

Mount-map rewriting (mount_map.py):
- per-container mount maps derived at rewrite time from inline CSI
  volumes the pod spec can name (GCS FUSE, Mountpoint-for-S3 ->
  gs://bucket / s3://bucket URIs, subPath-aware) plus an explicit
  [k8s.mount_map] config table for PVC-backed FUSE drivers whose bucket
  lives in the cluster-side PV; PVC mounts get pvc://claim identity
  tags and are never rewritten (their cross-pod edges connect through
  content hashes)
- injected as ROAR_K8S_MOUNT_MAP; pod_entry stamps the map into
  fragment metadata so capture stays raw and the mapping is auditable;
  collect_k8s_fragments rewrites ref paths at reconstitution with
  longest-prefix wins (streaming, bundle, and attach paths all covered)

Retry chaos e2e:
- a backoffLimit-1 Job whose first pod fails mid-run yields two
  attempt-distinct k8s_task jobs keyed by pod UID — one exit 1 with
  partial outputs, one exit 0 with final outputs, never conflated —
  proving the identity contract under real k8s pod recreation

Verified live on KIND: 16/16 e2e (chaos + mount-map new; mount-map via
config-declared map over a hostPath volume, asserting rewritten s3://
inputs/outputs and no raw mount paths leaking); 8 new unit tests;
default gate 2021 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…kend

Live operator validation:
- bootstrap --with-kubeflow installs training-operator v1 (v1.9.2) and
  trainer v2 (v2.2.1) + runtimes; --with-kuberay installs KubeRay v1.4.0
- PyTorchJob e2e proves the per-role adapter against the real
  controller (Master+Worker, RANK-chain node indices 0/1); TrainJob e2e
  runs the real TrainJob->JobSet pipeline via a slim-image clone of the
  shipped torch-distributed ClusterTrainingRuntime

RayJob delegation (rayjob.py, rewrite_style="rayjob"):
- entrypoint wrapped through the Ray driver entrypoint; runtimeEnvYAML
  merged with the roar pip requirement, worker setup hook, and the Ray
  env contract (ROAR_JOB_ID = the k8s parent uid so fragments link to
  the submit node); credentials only as Secret refs in the RayCluster
  pod templates; status.jobStatus terminal handling; reconstitution and
  attach delegate to the Ray backend's reconstituter (fragments are Ray
  TaskFragments merging as ray_task jobs)
- two live-found env-contract fixes: the merged local-proxy redirect
  (AWS_ENDPOINT_URL/ROAR_PROXY_PORT) is stripped (no proxy runs in the
  pods; user S3 traffic would hit a dead localhost port), and ROAR_WRAP
  stays off (the roar_inject.pth fires in Ray pip virtualenvs before
  site-packages are importable and the sitecustomize ABI-repair path
  blows the 60s worker registration timeout — observed as an infinite
  supervisor start/kill/retry loop)

Registry robustness (shared, found by the live smoke): backend plugin
imports that fail during early sitecustomize discovery were cached
forever ("unknown execution backend: ray" in worker setup hooks even
though deps import fine moments later); get_execution_backend now
retries skipped builtin imports on a lookup miss.

Known gap documented in the e2e and docs: per-task I/O fidelity inside
KubeRay workers under setup-hook-only bootstrap (fragments arrive as
ray_task shells without file refs) — a Ray-backend follow-up; the live
smoke asserts the delegation transport scope.

Verified live on KIND: 19/19 e2e (PyTorchJob + TrainJob + RayJob new);
10 new unit tests; default gate 2029 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Closes the per-task capture-fidelity gap on KubeRay found by the
delegation live smoke, with two root causes isolated by probing a real
worker (startup, open patch, executor patch, and task identity were all
verified engaged — yet fragments arrived empty):

- pathlib (Path.open/read_bytes/write_bytes) resolves io.open by module
  attribute, not the builtin, so the builtins-only open patch missed it;
  without the preload tracer (KubeRay pods) that I/O was invisible.
  roar_worker._startup now patches io.open alongside builtins.open. On
  preload-equipped native workers the duplicate python-level captures
  are absorbed by the existing native > python reconstitution dedup
- ray 2.46 does not route task execution through the patched
  FunctionActorManager.get_execution_info, so the per-task flush never
  engaged; the KubeRay e2e image is pinned to the native harness's
  ray 2.54 (rayproject/ray:2.54.0-py312-cpu) where the roar_worker
  internals are supported

The RayJob e2e's strict assertion (ray task file write present in
ray_task outputs) is restored and green.

Native-path regression check: identical compose-harness results with
and without this change (5F/1P parity runs; the failures are a
pre-existing runtime-env merge conflict on this dev box — job and
driver ray.init envs both carrying ROAR_NO_TELEMETRY under a host ray
2.54 CLI — tracked as a separate follow-up). Ray + k8s unit suites and
the full default gate are green.

Verified live on KIND: 19/19 e2e including the strict RayJob smoke;
default gate 2037 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Phase 3 slice 1: hermetic runtime staging (closes the Phase-0 "wheel is
not hermetic" finding — pods no longer need network installs).

- deploy/roar-runtime/Dockerfile: per-ABI dependency trees (cp310-cp313,
  resolved by uv --python-version without needing each interpreter),
  each with a generated top-level sitecustomize.py mirroring
  roar_inject.pth (PYTHONPATH staging doesn't process .pth files); the
  image doubles as the staging init container (default CMD copies
  /opt/roar-runtime into the shared mount) and the webhook server base
- k8s.runtime_source=image + k8s.runtime_image config: the rewriter
  injects a roar-runtime-staging init container + emptyDir volume
  (idempotent by name for webhook reinvocation) and the wrapper
  activates the ABI-matched tree via PYTHONPATH instead of pip;
  TrainJob keeps the install path (no inline pod template for an init
  container); RayJob keeps its runtime-env pip mechanism
- scripts/build_runtime_image.sh; .dockerignore wheel exception

Verified live on KIND: image-mode e2e green in ~11s (vs ~16s+ for pip
mode) — staging init container present, no pip in the wrapper, lineage
captured; 4 new unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Phase 3 slice 2: the platform-install path. A stdlib HTTPS
AdmissionReview server (roar/backends/k8s/webhook.py, served by the
roar-runtime image) intercepts CREATE of all five workload kinds in
namespaces labeled roar.glaas.ai/lineage=enabled and applies the same
manifest rewrite as the CLI path — clients need no roar at all.

- per admission: mints the fragment session against GLaaS, creates the
  credentials Secret through the k8s API (never embedded in the
  object), rewrites via the shared rewriter (image-staged runtime by
  default), annotates the workload (roar.glaas.ai/parent-uid,
  session-id, fragment-secret), returns a JSONPatch
- never blocks admission: every internal failure returns allowed with
  a warning, paired with failurePolicy Ignore; dry-run admissions are
  side-effect-free (sideEffects NoneOnDryRun); idempotent under
  reinvocationPolicy IfNeeded via the parent-uid annotation
- reconstitution is client-driven via the existing roar k8s attach
  (reads the webhook-created Secret)
- harness: bootstrap --with-webhook builds/loads roar-runtime:dev and
  deploys via deploy_webhook.sh (self-signed CA, RBAC for secret
  creation, Deployment/Service/MutatingWebhookConfiguration); the
  image is always rebuilt (layer cache) — a stale image silently
  breaks the webhook
- fix found live: the API server posts /mutate?timeout=10s — path
  matching must strip the query string

Verified live on KIND: 22/22 e2e — zero-touch test does a plain
kubectl apply in a labeled namespace (no roar client), asserts
injection (annotations, staging init container, wrapped command), job
completion, and attach-recovered lineage keyed to the webhook-minted
parent uid; unlabeled-namespace control untouched. 7 new unit tests;
default gate 2040 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…executors

Closes the ray 2.46 circle-back — by disproving it. A local probe
(worker_process_setup_hook wrapping FunctionActorManager methods)
showed ray 2.46 DOES route task execution through get_execution_info;
the empty task fragments previously blamed on 2.46 were entirely the
pathlib/io.open capture gap, confounded because that fix landed
together with the 2.54 image pin, which took the credit.

- strict KubeRay e2e verified green on BOTH 2.46.0 and 2.54.0 with the
  final code; the e2e image is now overridable via ROAR_E2E_RAY_IMAGE
  (default stays the native harness's 2.54 pin) and rayVersion in the
  RayJob template tracks the image tag
- detect-and-warn insurance: _patch_ray_task_execution_for_native_flush
  now prints a loud stderr warning (with the ray version) when the
  function manager cannot be imported or exposes neither
  get_execution_info nor _make_actor_method_executor, instead of
  silently degrading to task fragments without identity or file refs
- developer doc and design doc corrected

Verified: strict RayJob e2e 1/1 on 2.46 and 1/1 on 2.54; ray + k8s
unit suites green; default gate 2040 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add k8s.proxy_sidecar / k8s.proxy_upstream (and webhook
ROAR_WEBHOOK_PROXY_SIDECAR / ROAR_WEBHOOK_PROXY_UPSTREAM): the rewriter
appends a roar-s3-proxy native sidecar (init container with
restartPolicy Always) running the roar-proxy binary from the staged
runtime tree, so S3 traffic from clients the botocore hooks cannot see
(non-Python binaries, raw HTTP) still lands as s3:// lineage refs.

- Requires image staging; install mode warns and skips.
- roar-proxy re-signs upstream requests, so the sidecar inherits the
  workload container's AWS_* credential env entries.
- The proxy binds loopback only; the startup probe is an exec probe
  (kubelet tcpSocket probes dial the pod IP and never succeed).
- Wrapped containers get AWS_ENDPOINT_URL=http://127.0.0.1:19191 only
  when the user has not set their own (explicit endpoints win).
- pod_entry parses the proxy log (shared emptyDir) and folds refs into
  the fragment with capture_method="proxy", deduping objects the hooks
  already captured.
- Image-mode fragments no longer report the staged /roar-runtime/**
  tree as inputs (filtered at reconstitution; capture stays raw).

Live-validated on the KIND harness with a hook-invisible raw-urllib S3
client against MinIO (23/23 k8s e2e).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add deploy/charts/roar-lineage-webhook: webhook Deployment/Service,
RBAC (Secret-create ClusterRole), and the MutatingWebhookConfiguration
for all five workload kinds. TLS comes either from a supplied secret +
tls.caBundle (required-value guarded) or from an optional cert-manager
self-signed Issuer/Certificate with CA injection. injection.* values
map 1:1 onto the ROAR_WEBHOOK_* env contract, including the proxy
sidecar knobs.

The harness deploy script now generates self-signed TLS material and
installs this chart (helm downloaded into .tools/bin like kind/kubectl),
so the zero-touch webhook e2e proves the packaged artifact end-to-end;
the e2e webhook-presence check selects the configuration by chart labels
instead of the old raw-manifest name.

Validated: helm lint clean, cert-manager and manual-TLS renders, and
23/23 k8s e2e against the chart-deployed webhook.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Long training jobs routinely outlive the fragment-session TTL, and an
expired session 403s on both append and read — lineage was lost with no
recovery path. Pair with the glaas-api renewal endpoint
(POST /api/v1/fragments/sessions/:id/renew, token-authenticated and
allowed even after expiry):

- GlaasFragmentStreamer renews on a 403 batch POST and retries the
  chunk once; a second 403 gives up normally. Renewal is purely
  reactive — no client-side expiry clock.
- Both fragment reconstituters (k8s, ray) renew-and-retry on a 403
  fetch, so a late `roar k8s attach` recovers an expired session too.
- GlaasClient.renew_fragment_session for host-side callers.
- Streamer exposes delivered/failed/pending counters and close()
  reports flush success (used by the follow-up transport change).

Degrades gracefully against a server without the endpoint (renew 404s
-> previous behavior). Live-proven on the KIND harness: 60s-TTL session,
75s job, streamer renews mid-run and the finalizer reconstitutes
(tests/backends/k8s/e2e/test_k8s_ttl_renewal.py; skips when the local
glaas-api lacks the endpoint).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

emit_fragment_dicts returned "streamed" whenever credentials were
present, even when every batch POST failed — mid-run streaming failures
were silently lost because callers (k8s pod_entry bundle fallback, ray
local merge) only fall back on a non-"streamed" result. It now returns
"streamed" only when the streamer delivered everything; partial or
total failures log an undelivered-count warning and fall through to
local_merge / local_fallback with all fragments (downstream merges
dedupe by task identity, so overlap with delivered batches is
absorbed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…me envs

Every `roar run ray job submit` whose driver called ray.init() died with
"Failed to merge the Job's runtime env ... because of a conflict": the
submit rewrite delivers ROAR_NO_TELEMETRY (plus the rest of the env
contract) in the Job-level runtime env, and the patched driver-side
ray.init re-added ROAR_NO_TELEMETRY — ray >= 2.4 rejects the Job/driver
merge on ANY duplicated env-var key, even with identical values. The
failure was roar conflicting with itself, not a ray version issue.

The instrumented driver path now reads the job config ray injects
(RAY_JOB_CONFIG_JSON_ENV_VAR) and only adds what the Job env lacks
(same for py_executable — a job-level value wins with a warning). The
non-instrumented path gets a generic guard that drops roar-added env
keys the Job env already carries; user-supplied keys are left alone so
genuine conflicts still surface through ray's own error. The merged
runtime env that reaches workers is unchanged.

Live-validated on the compose harness: the previously-failing
runtime-env-conflict and native-tracing e2e pass, and the suite goes
from dead-on-arrival instrumented submits to 57 passing (remaining
failures are pre-existing capture-fidelity gaps newly reachable now
that submits run, tracked separately).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The streamer splits an oversized fragment into parts sharing one task
identity after an HTTP 413; the keep-last-per-identity dedup then
silently discarded every part but the last, losing most of a large
task's reads/writes. Retries are already attempt-distinct in the
identity contract (pod uid + restart attempt), so same-identity
fragments can only be duplicate deliveries or split parts — dedup now
unions reads/writes per path across them (last ref wins, last
fragment's scalars win). No wire-format change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ctl global flags

Both submit planners saved the fragment-session key (and k8s the
prepared manifest) under a bare cwd/.roar, while the run finalizer loads
the key from the context-resolved project root — submitting from a
project subdirectory created a stray nested .roar and lineage silently
never reconstituted locally. Planners now resolve the project .roar via
the same upward walk the CLI context uses (ROAR_PROJECT_DIR override,
bounded by the git root), shared as
roar.execution.fragments.sessions.resolve_project_roar_dir. Covered by
unit tests for both backends and a KIND e2e that submits from a nested
directory.

The kubectl matcher also accepted the verb only at argv[1], so common
forms like `kubectl --context prod apply -f job.yaml` silently bypassed
instrumentation. The verb now matches anywhere; the existing
-f-manifest and single-supported-workload guards keep false positives
out.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The RayJob rewrite set ROAR_JOB_ID to the k8s parent uid but never
passed driver_job_uid to merge_worker_bootstrap_env, so workers had no
ROAR_DRIVER_JOB_UID and their fragments merged without a parent —
ray_task rows carried no DAG edge back to the k8s submit job (the merge
prefers the fragment-level parent, which was empty, and the host-side
fallback env is unset in this flow). The live RayJob e2e now asserts
every ray_task row parents to the recorded submit job, which is exactly
the assertion whose absence let this slip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…et cleanup

The admission handler registered the GLaaS session and created the
credentials Secret before attempting the manifest rewrite, so a
rewrite failure (the step most likely to fail) orphaned both. The
rewrite — a pure function of the admitted object — now runs first;
external resources are only created once it succeeds.

Credential Secrets can never carry an ownerReference (the workload has
no UID at CREATE admission), so the chart gains a secretCleanup
CronJob (roar.backends.k8s.secret_cleanup, runs on the runtime image)
deleting roar-fragment-* Secrets older than the maximum session TTL.
It uses its own ServiceAccount with list/delete only; the webhook's
stays create-only. Running pods are unaffected (env resolves at pod
start) — only late attach credential recovery ages out with the
session itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The pod wrappers guarded python3/pip/staged-tree presence but exec'd
pod_entry unconditionally — an import failure past exec (broken
install, ABI mismatch such as the x86-64 staged tree on an arm64 node,
unsupported interpreter) killed the container with no way back. An
import probe now runs as the last step before exec and falls back to
the original command uninstrumented, which also makes non-x86-64
platforms degrade gracefully instead of failing training. The
remaining unisolated window (a crash inside pod_entry after the probe)
is documented instead of overclaimed.

Also adds the missing rayjob/rayjobs attach aliases (a
WORKLOAD_KINDS-completeness test keeps future kinds honest) and
mentions rayjob in the attach help text.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The RayJob rewrite destroyed three kinds of user runtime configuration:
pip dicts were flattened to package lists (losing pip_check/pip_version),
a user worker_process_setup_hook was silently replaced with roar's, and
a user-supplied AWS_ENDPOINT_URL (MinIO, LocalStack, private S3) was
stripped along with roar's dead local-proxy redirect.

- keep the pip dict shape and append the roar requirement to packages;
  reject pip requirements-file references with an actionable error
- carry a displaced user setup hook as ROAR_USER_SETUP_HOOK; roar's
  worker_bootstrap.startup chains it after capture installs, and user
  hook failures propagate exactly as they would uninstrumented
- drop the local-proxy redirect from the merge source instead of popping
  the merged result, so user endpoints survive the rewrite

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

pod_entry returned roar run's exit code directly, so a tracer/runtime
failure before the workload launched failed the training job — breaking
the documented lineage-never-blocks-training contract (ARM nodes and
unsupported native boundaries were the likely triggers).

roar run already distinguishes pre-launch setup failures from workload
exits (ExecutionReport.setup_error: nonzero exit with no job created).
Expose it across the subprocess boundary: when ROAR_RUN_REPORT_FILE is
set, the run service writes {exit_code, setup_error, tracer_backend}
before finalizers run. pod_entry reruns the original command
uninstrumented only on a positively reported setup failure — a missing
or unreadable report is ambiguous (roar may have died after the
workload ran) and must propagate rather than risk double-running
non-idempotent training.

New KIND chaos e2e forces the failure with k8s.tracer=ebpf (valid mode,
cannot pass preflight in an unprivileged pod) and proves the Job still
completes with the workload's own outcome and no fabricated lineage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Bundle fallback derived its filename from the pod name alone, so two
wrapped containers in one pod (or an in-place restart) overwrote each
other's fragments last-writer-wins. Bundle names now carry
pod-container-attempt identity (attempt resolution mirrors the task_id
contract) and are written atomically via os.replace so ingest never
reads a torn bundle.

The submit matcher also accepted commands with several -f/--filename
arguments but rewrote only the first: one workload got lineage and
completion polling while the rest were applied unrewritten and
unwaited. Multiple manifest arguments now decline k8s instrumentation
with an actionable warning and leave the kubectl command untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Session .key files carry the raw fragment token but were written with
write_text, inheriting the umask (0664 observed) — group/other could
read the credential. Keys are now created 0600 from the first byte and
swapped in with os.replace so a rewrite of an existing over-permissive
key also ends owner-only.

Also removes the unused GlaasClient.renew_fragment_session, which put
the raw token in the URL query string where access logs and proxies
capture it. The live renewal path (fragment_streamer) authenticates
with the x-roar-fragment-token header and stays.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The cleanup CronJob defaulted on, granting its ServiceAccount
cluster-wide list/delete on Secrets — list responses include Secret
data, so a default install was one compromised cleanup image away from
cluster-wide credential read. Its fixed-age policy also conflicts with
session renewal: a renewed workload outliving maxAgeSeconds loses the
Secret its retry/scale-up pods reference, and attach recovery with it.

The CronJob is now opt-in, with both trade-offs documented in
values.yaml, the module docstring, and the developer doc. Liveness-aware
(controller-grade) cleanup remains deliberately deferred.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The verb-anywhere matcher accepted global-flag VALUES as the verb:
'kubectl --context apply delete -f job.yaml' and
'kubectl delete -f job.yaml --namespace create' both matched the submit
backend, which would register a session, rewrite a manifest being
deleted, and wait on a workload that no longer exists.

The matcher now finds the first non-flag token after skipping known
value-taking kubectl global flags (space and = forms); both review
commands are regression tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The Dockerfile globs dist/, so multiple wheels silently produced an
image containing whichever sorted first — typically a stale build. The
build script now fails with the wheel list and prints which wheel a
successful build used.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Managed capture requires host-side GLaaS at submit time; bundle_dir
covers pod-side outages only. Documented explicitly (with the prepare
escape hatch) until the deferred local-first design lands, and the
match description updated for structural verb parsing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

pod_entry kept .roar/roar.db, object-io events, and the run report in
the workload's workdir. On a shared or persistent volume (emptyDir
shared by co-located wrapped containers, PVC surviving retries) every
participant shared one sqlite db and one events file — racing writes
and cross-attributing lineage between containers and attempts.

roar state now lives in pod-local storage keyed by the identity
contract (pod_uid:container:completion_index:restart_attempt); the
workload keeps its own cwd. RoarContext.create now honors an existing
ROAR_PROJECT_DIR — the contract config loaders and fragment planners
already follow — gated on the directory existing so Ray-propagated host
paths inside pods never resolve to a phantom project. An unusable state
dir falls back to an uninstrumented run rather than failing training.

New shared-workdir KIND e2e: two wrapped containers on one emptyDir
volume reconstitute as two distinct k8s_tasks, each owning only its own
outputs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The tracking import hook routes every import — including the retry's own
plugin imports — through handle_import, whose backend resolution lands
back in get_execution_backend. Re-entering the retry there fans each
hooked import inside each plugin module out into another full retry
pass: unbounded mutual recursion that pins the CPU and keeps Ray workers
from ever registering (observed at worker startup on the K8s/Ray dogfood
cluster, py-spy stack on file).

Guard the retry with an in-progress flag and skip it while discovery is
running; nested lookups fall through to LookupError, which hook callers
already treat as 'no backend yet'. Regression test fails pre-fix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

S3 streams CompleteMultipartUpload responses as Transfer-Encoding:
chunked whitespace keep-alives. The proxy buffers that body (for ETag
extraction) but copied the upstream framing headers verbatim, producing
a fixed-length response that claims chunked framing — hyper drops the
connection and clients see 'empty reply from server' (curl 52) on every
multipart completion. Let hyper frame the body itself.

Verified A/B against a chunked fake upstream: pre-fix curl exit 52,
post-fix 200 with intact body.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The task-executor wrapper was always synchronous, so wrapping an async
actor method (Ray Train v2 TrainController.run) hid its coroutine-ness
from Ray's dispatch: the un-awaited coroutine became the task result and
failed to pickle ('cannot pickle coroutine object'). Wrap coroutine
functions with an async wrapper that awaits under the same bookkeeping
scope.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

On a fresh clone with no .roar directory anywhere, the submit planner
running in the workflow's working_directory subdir fell back to
<subdir>/.roar while the run finalizer (repo-root cwd) loads from
<root>/.roar — the fragment session key was stranded and lineage
publication failed even after a fully green training run.

Behavior change: the documented no-project fallback inside a git repo is
now <git-root>/.roar (matches CLI project-context semantics).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>